Repository navigation
Map OpenMP reduction variables explicitly on NVHPC - #2018
Conversation
Since MFlowCode#1821, the Phoenix NVHPC gpu-omp lane intermittently fails 3D -> 2 MPI Ranks -> IBM Sphere (AFBACA70) and 3D -> 2 MPI Ranks -> IBM Moving Sphere -> ib_neighborhood_radius=2 (2440B4AE) with "call to cuMemcpyDtoHAsync returned error 700 (CUDA_ERROR_ILLEGAL_ADDRESS)" at step 1, on master and unrelated PRs. It fails whenever the two ranks see more than one GPU, so rank 1's OpenMP default device is 1; it passes when they share a GPU, which is why retries sometimes succeed. compute-sanitizer pins it to an out-of-bounds atomic at the end of the s_ibm_correct_state ghost-point loop (m_ibm.fpp:598, launched at :345): the reduction(+:n_not_converged, n_ill_posed) that MFlowCode#1821 added. A standalone nvfortran 25.11 program reproduces it outside MFC: a multi-variable scalar reduction on a target teams loop run on default device 1 fails with CUDA_ERROR_ILLEGAL_ADDRESS when device 0's CUDA context is current (as UCX leaves it after MPI_Init) and with CUDA_ERROR_INVALID_CONTEXT when none is. NVHPC allocates the implicitly mapped reduction storage in the current CUDA context instead of on the default device. Mapping the reduction variables explicitly with map(tofrom:) avoids it; m_bubbles_EL already pairs its two reductions with copy= of the same variables. Emit map(tofrom:) for the reduction variables in OMP_PARALLEL_LOOP on NVHPC, skipping any already listed in copy. This covers all six reduction loops without one (m_chemistry, m_collisions, m_data_output, m_ibm, m_pressure_relaxation, m_time_steppers). Other compilers and OpenACC are unchanged: the Fypp output is identical apart from those six directives. Verified on 3x A100 with NVHPC 25.11 (--gpu mp, MPI, release): - master: the six 2 MPI Ranks tests with -g 0 1 give 4 passed, 2 failed (AFBACA70, 2440B4AE: memory access fault by GPU); direct runs with both GPUs visible fault 5 of 5 times. - this change: 6 of 6 pass, three times; the full suite with -g 0 1 2 gives 722 passed, 0 failed.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Lines of Code
|
|
Claude Code Review Head SHA: 615eabf Files changed:
Findings: |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2018 +/- ##
=======================================
Coverage 61.75% 61.75%
=======================================
Files 86 86
Lines 22787 22787
Branches 3357 3357
=======================================
Hits 14073 14073
Misses 6225 6225
Partials 2489 2489 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Since #1821, the Phoenix NVHPC gpu-omp lane intermittently fails
3D -> 2 MPI Ranks -> IBM Sphere (AFBACA70) and
3D -> 2 MPI Ranks -> IBM Moving Sphere -> ib_neighborhood_radius=2 (2440B4AE)
with "call to cuMemcpyDtoHAsync returned error 700 (CUDA_ERROR_ILLEGAL_ADDRESS)"
at step 1, on master and unrelated PRs. It fails whenever the two ranks see
more than one GPU, so rank 1's OpenMP default device is 1; it passes when they
share a GPU, which is why retries sometimes succeed.
compute-sanitizer pins it to an out-of-bounds atomic at the end of the
s_ibm_correct_state ghost-point loop (m_ibm.fpp:598, launched at :345): the
reduction(+:n_not_converged, n_ill_posed) that #1821 added. A standalone
nvfortran 25.11 program reproduces it outside MFC: a multi-variable scalar
reduction on a target teams loop run on default device 1 fails with
CUDA_ERROR_ILLEGAL_ADDRESS when device 0's CUDA context is current (as UCX
leaves it after MPI_Init) and with CUDA_ERROR_INVALID_CONTEXT when none is.
NVHPC allocates the implicitly mapped reduction storage in the current CUDA
context instead of on the default device. Mapping the reduction variables
explicitly with map(tofrom:) avoids it; m_bubbles_EL already pairs its two
reductions with copy= of the same variables.
Emit map(tofrom:) for the reduction variables in OMP_PARALLEL_LOOP on NVHPC,
skipping any already listed in copy. This covers all six reduction loops
without one (m_chemistry, m_collisions, m_data_output, m_ibm,
m_pressure_relaxation, m_time_steppers). Other compilers and OpenACC are
unchanged: the Fypp output is identical apart from those six directives.
Verified on 3x A100 with NVHPC 25.11 (--gpu mp, MPI, release):
(AFBACA70, 2440B4AE: memory access fault by GPU); direct runs with both
GPUs visible fault 5 of 5 times.
722 passed, 0 failed.
Also checked with NVHPC 24.5 (nvcr.io/nvidia/nvhpc:24.5-devel-cuda_multi-ubuntu22.04, the CI image, --gpu mp, MPI, 2x A100 visible):
"HX_CU_CALL_CHECK(p_cuStreamSynchronize(stream[dev]))" rather than CUDA_ERROR_ILLEGAL_ADDRESS.
Acknowledgement